Adaptive Load Balancing in MapReduce using Flubber

MapReduce has emerged as a successful framework for addressing the heavy demand for large-scale analytical data processing, in this peta-byte age. However, while on one hand the sheer size of data makes problems more challenging, the flexibility offered by the MapReduce frameworks on the other hand, makes the learning curve far steeper than expected. The general idea behind a MapReduce framework is to split the task into two components – a Mapper and a Reducer. The mapper executes a user-defined computation on chunks of data and generates the results, while the reducer groups the results together based on a common attribute. Scalability, hence, appears as an inherent trait of the design. A critical parameter in this configuration is the number of reducers required for a given task, and frameworks like Hadoop expect the user to specify this parameter while submitting a job. In this report, we focus on Hadoop and argue that deciding the number of reducers is a non-trivial task, let alone deciding it prior to running the job. To address this issue, we present Flubber – a simple pre-job that can be sandwiched between the original job and Hadoop. With a couple of parameters from the user, it takes a stab at figuring out the ideal number of reducers for the given job.